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Using a functional renormalization group approach we study the zero temperature phase diagram 
of two-dimensional Bose-Fermi mixtures of ultra-cold atoms in optical lattices, in the limit when the 
velocity of bosonic condensate fluctuations are much larger than the Fermi velocity. For spin- 1/2 
fermions we obtain a phase diagram, which shows a competition of pairing phases of various orbital 
symmetry (s, p, and d) and antiferromagnetic order. We determine the value of the gaps of various 
phases close to half-flUing, and identify subdominant orders as well as short-range fluctuations from 
the RG flow. For spinless fermions we flnd that p-wave pairing dominates the phase diagram. 
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Since the realization of the Mott insulator (MI) tran- 
sition Q in ultra-cold atom systems, there has been re- 
markable progress in 'engineering' many-body states in 
a well-defined and tunable environment, due to the ad- 
vances in trappingand manipulating ultra-cold atoms in 
optical lattices Q ■ Important achievements include 
the creation of fermionic superfluids j^, the realization 
of the Tonks-Girardeau gas and of Luttinger liquids 
0, and the observation of noise correlations One 
of the subjects of intense study recently has been mix- 
tures of ultra-cold bosonic and fermionic atoms. Experi- 
ments include the condensation of molecules in mixtures 
of fermionic atoms '3| , and the observation of instabilities 
in Bose-Fermi mixtures (BFM) j9|. Many other intrigu- 
ing many-body phenomena have been proposed, such as 
the appearance of ch arge density wave order (CDW) [l^, 
de-mixing transition , the formation of composite par- 
ticles jl2], and polaronic effects 0. Mixtures of ultra- 
cold bosonic and fermionic atoms that are subjected to 
optical lattices, which confine the atoms to move on a 
2D lattice, exhibit one of the most intriguing phenomena 
of condensed matter physics, the competition of orders 
in many-body systems, which is also observed in a va- 
riety of materials, s^uch as high-Tc compounds Q| and 
Bechgaard salts [l^- To capture the properties of the 
many-body state of these systems presents a significant 
theoretical challenge. 

In this paper we study these systems using a functional 
renormalization group (RG) approach introduced in the 
context of the stability of the Fermi liquid fixed point 
|l6| , and that has been extensively applied to interacting 
electrons on lattices in the last few years ^3, and 
has recently been extended to include retardation effects 
associated with the electron-phonon interaction ;19» .2(11 . 
Contrary to mean- field and variational approaches |2l| . 
the RG approach includes corrections to the full four- 
point vertices in the flow equations, and therefore treats 
all types of order in an unbiased way. In this way we ob- 
tain the phase diagram and the values of the gaps of the 



different types of order. Furthermore, we can read off the 
subdominant orders and the short distance fluctuations 
from the RG analysis. 

The fermionic atoms in the BFM considered here are 
prepared either as a mixture of two hyperfine states 
(which we treat in an isospin language), to create spin- 
1/2 fermions, or in a single hyperfine state to create 
spinless fermions. Fermionic atoms in different hyperfine 
states interact via short-range, i.e., on-site interaction, 
whereas spinless fermions are essentially non-interacting. 
Besides two-body contact interactions, density fluctua- 
tions in a condensate of bosonic atoms induce attractive 
finite-range interactions between fermions, with a length 
scale given by the coherence length of the condensate. 
The competition between these two types of interaction 
leads to the possibilities of many types of different in- 
stabilities and, hence, to a rich phase diagram, as we 
discuss here. In the laboratory, this system can be real- 
ized as a ''°K-^'^Rb mixture in an optical lattice created 
by NdiYAG lasers. The interaction between the differ- 
ent atomic species can be manipulated by either tuning 
the system close to a Feshbach resonance, or by using 
more than one optical lattice to trap the different types 
of atoms and to spatially shift these lattices with respect 
to each other. 

Ultracold atoms in optical lattices are very accurately 
described by a Hubbard model. In the following we write 
the model for the case of spin-1/2 fermions. The case of 
spinless fermions can be immediately obtained from this 
by ignoring one of the spin states. For a mixture of one 
type of bosonic atom and two fermionic types that are 
SU(2) symmetric, the Hamiltonian is given by: 

+ ^ [t^//'^/.i,T"-/,ia + -Y'^b,inb,i + UbfUb^iUf^i ,(1) 
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where //^ ifi.s) creates (annihilates) a fermion at site 
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FIG. 1: Phase diagram, interaction strength, Uff/tf, versus 
number of fermions per site, n, for a Fermi-Bose mixture in a 
square lattice in 2D {V/tf = 2, ^ = 1). 



i with pseudo-spin s (s =t,J-), &| (^i) creates (annihi- 
lates) a boson at site i, nj^i = Es fi,sfi,s = ^1^0 is 
the fermion (boson) number operator, tf and i;, are the 
fermionic and bosonic tunneling energies between neigh- 
boring sites, p f (fj,h) is the chemical potential for fermions 
(bosons), Ubb is the repulsion energy between bosons on 
the same site, Uff is the interaction energy between the 
two species of fermions, and Ubf is the interaction energy 
between bosons and fermions. In momentum space, this 
Hamiltonian is written as: 

H = ^< (e/,k-/^/)X!/L/k,s + (eb,k-A*b)&k&k 

k I s 
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where Pf^^, = Eq,. /k+q^Jq,^ (PbM = Eq^k+q^q) 

is the fermion (boson) density operator, ^b/f.it. — 
—2tij/f{coskx + cosfcy), is the bosonic/fermionic disper- 
sion relation for free particles on a square lattice. 

Throughout this paper we consider the limit of weakly 
interacting bosons that form a BEC. We assume further 
that the velocity of the condensate fluctuations is much 
larger than the Fermi velocity. These two assumptions 
allow us to integrate out the bosonic degrees of freedom 
and use an instantaneous approximation, i.e. neglect 
retardation effects. In the limit of weakly interacting 
bosons, the following approach is well established: We 
assume that the zero momentum bosonic mode is macro- 
scopically occupied, and we formally replace the corre- 
sponding operator bo by a real number bo ~> VNq, where 
No is the number of condensed atoms. After this re- 
placement we keep all terms that are quadratic in 6k 
(with k 7^ 0), and perform a Bogoliubov transforma- 
tion, given by: 5k — Mk/3k + Wk/^lk, ^'^ diagonalize the 
bosonic Hamiltonian. The resulting eigenmodes /3k have 
a dispersion relation given by Wk ~ ■\/eb,k(e6,k + 2Ubbnb), 
with the low-k limit cjk ^ Vb\^\-, with Vb = \^2tbUbbnb- 



eb,k + Ubbnb)/{2uJk) and = (-Wk + eb,k + t^bb"-b)/(2wk)- 
The density fluctuations of the bosons are approximated 
by: Pb,k ~ %/^(wk + Vk)(/3k + /3lk), with k 7^ 0. 
The interaction between bosons and fermions is given by 

C^6/V^/^Ek("k + ^^k)(/3k+/3lk)/^/-k- As a next step 
we integrate out the bosonic modes and use an instan- 
taneous approximation, leading to the following effective 
Hamiltonian: 
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where the induced potential V^nd.,k is given by: yind.,k — 
~V/{1 + ^^(4 - 2cosA:a; - 2cosfcy)) , with V given by 
V — U^j/Ubb, and ^ is the healing length of the BEC and 

is given by f = ^t\^j2n\,Ubb- Notice, that this approach 
is only valid when Uf, 3> w/, so that the fermion- fermion 
interaction mediated by the bosons can be considered as 
instantaneous. In the presence of retardation one can- 
not formally define a Hamiltonian formulation since the 
frequency dependence of the interaction appears explic- 
itly. If that is the case, one has to consider the frequency 
dependence of the interaction explicitly as is done, for 
instance, in ref. jl^]. Here, however, the system can be 
tuned to the non-retarded limit, that is not the limit in 
most solid state systems. 

Notice that Q describes the scattering of two fermions 
from momenta ki and k2, that are scattered into mo- 
menta ka and k4. Momentum conservation at the inter- 
action vertex requires that k4 = ki -I- k2 — ka , and hence 
the interaction vertex, J7(ki,k2,k3), depends on three 
momenta 1231 . Its bare value can be written as: 



C/(ki,k2,k3 



ind..ki — k3 



(4) 



The RG is implemented by tracing out high energy de- 
grees of freedom in a region between A and A-FdA, where 
A is the energy cut-off of the problem. In this process, 
the vertex \J is renormalized. At the initial value of the 
cut-off A = Aq « 8i/, the value of \J is given by Q), 
which is the initial condition for the RG. The RG flow 
is obtained from a series of coupled integral-differential 
equations 0| for all the different interaction vertices 
?7(ki, k2, ka). From these, the specific interaction chan- 
nels, such as CDW, antiferromagnetic (AF), and super- 
conducting (BCS), can be identified: 

yCDW ^ 4[/,(ki,k2,ki+Q), 

V^F = 4 [/,(ki,k2,ki + Q), 



(5) 
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The parameters Uk and Wk are given by: 



(Wk 



where we have use used: Uc = (2- A')C//4, [/g. = -XU/A 
with A't/(ki, k2, ka) = C/(k2, ki, ks), and Q is the nest- 
ing vector, Q — (7r,7r). The RG equations read: 

9f[/£(ki,k2,k3) = 
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- / a, [G, (p)G, (k)] t/, (ki , k2 , k) [/, (p, k, ka ) 
-( 5, [G, (p) G£ (qi )] t/, (p, k2 , qi ) t/, (ki , qi , ka ) 

J p,UJ 

- j 5^[G^(p)G,(q2)] {-2C/^(ki, p, q2)C/^(q2, k2, kg) 

J p,iJ 

+Ue(p,ki ,q2) Ue (q2 ,k2 ,k3) + Ue (ki ,p,(i2)Ue (k2 ,q2 ,k3)},(8) 

where £ = ln(Ao/A), k = ki + k2 - p, qi = p + k2 - ks, 
q2 = p + ki - k3, and G,(k) = e(|Ck| - A)/(zc^ - fk) 
with = e/,k - fJ.f- 

In our implementation we discretize the Fermi surface 
into M = 24 patches, and hence each of the interaction 
channels (O, ©, ©is represented by an M x M matrix. 
At each RG step, we diagonahze each of these matrices. 
The channel with the largest eigenvalue (with the caveat 
that a BCS-channel needs to be attractive to drive a tran- 
sition) corresponds to the dominant order. The elements 
of the eigenvector are labeled by the discrete patch in- 
dices around the Fermi surface and the symmetry of the 
order parameter is given by this angular dependence. In 
some parts of the phase diagram we encounter a diver- 
gence in the RG flow, indicating the onset of ordering 
with a gap that is in the detectable regime, i.e. larger 
than 10~^tf. In other regimes, where such a divergence 
is not reached, one can read off the dominant tendency 
of the RG flow. In Fig. |21we show examples of RG flows 
as a function of ^. In Fig. [51 (a), we show the competition 
between d-wave and s-wave pairing, with d-wave being 
dominant and s-wave being subdominant. In Fig. [21 (b) 
we show an example with dominant d-wave channel and 
subdominant AF channel. In both cases we find that for 
short distances (or high energies) CDW fluctuations are 
dominant, giving rise to a state that resembles the find- 
ings for high- Tc superconductors. Note that in some sit- 
uations the many-body states are almost degenerate and 
small changes in the initial conditions (that is, changes 
in the form of the interactions) can be used to select one 
particular ground state. 

With this procedure we determine the phase diagram 
of the system, which is shown in Fig. ^ We now dis- 
cuss the general features of the phase diagram. In the 
absence of any coupling to the bosons, i.e. for V = 0, the 
system shows s-wave pairing for attractive interaction, 
Uff < 0, and no ordering for > 0, i.e. Fermi liquid 
behavior, except for the special case of half-filling where 
Fermi surface nesting drives the system to AF order for 
repulsive interactions, and to s-wave pairing (degenerate 
with CDW) for attractive interaction. If we now turn on 
the interaction to the bosons, this picture is modified in 
the following way: The boundary of the s-wave regime 
is moved into the regime of positive Uff, approximately 
to a value oi Uff where the effective interaction at the 
nesting vector Q between the fermions, Uff + 14nd.,Q, is 
positive, i.e. for Uff ^^/(l + 8^^). On the repulsive 
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FIG. 2: RG flow for the difi'erent efi'ective interactions (in 
units of tf) as a function of the RG parameter I {V/tf — 3 
and 5 = 1). a) Uff ftp = 0.5; b) Uff/tf = 1.2. 



side, and away from half-filling, we find the tendency to 
form a paired state, either d-wave or p-wave. This ten- 
dency becomes weaker the further the system is away 
from half-filling. We typically find a gap in the vicin- 
ity of half-filling and further away from /i = we find 
only an increasing strength of the corresponding inter- 
action channel. For the half-filled system, we find that 
for attractive interactions the degeneracy between s-wave 
pairing and CDW ordering is lifted, with s-wave pairing 
being the remaining type of order. For repulsive interac- 
tions, we find an intermediate regime of d-wave pairing, 
and for larger values of Uff we obtain AF order. 

The RG approach also allows the extraction of the 
many-body gaps in the system through a "poor man's 
scaling" analysis of the divergent flow: at the point where 
the coupling becomes of order of t f the scaling parameter 
£ reaches the maximum value £* = ln(t//A), where A is 
the value of the gap. Hence, A/f/ w exp{— £*} can be 
obtained from the RG flows such as the ones in Fig. [3 
In Fig. [31 we show the gaps of the problem as a function 
of Uff/tf in the half-filled case. One can see that as Uff 
increases, from negative to positive values, the s-wave 
gap is replaced by a d-wave gap, and finally for an anti- 
ferromagnetic gap. As is apparent from this figure, the 
gap in the d-wave phase is much smaller than the gaps of 
the AF order and the s-wave pairing, and, furthermore, 
almost independent of the value oiUff. The latter is the 
case because the Uf / term is a pure s-wave contribution 
to the interaction and therefore does not contribute to 
the d-wave channel. The d-wave channel has an initial 
contribution which is entirely due to the anisotropy of 
the induced interaction, which gives only a small value, 
and as a consequence only a small value for the gap. The 
value of the gap (in units of t/) can be numerically fitted 
with a BCS expression of the form a cxp(— 6/y), with the 
parameters a and b given by a = 0.31 and b — 14.2. 

We have also performed RG calculations for a system 
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We found a number of competing phases, in particular in 
the vicinity of half- filling, including AF ordering, s-, p-, 
and d-wave pairing. Our approach enables us to quantify 
the gaps of these phases, to identify subdominant orders, 
and to study short-range fluctuations. Optical lattices 
of cold atoms allow for a unique opportunity of study 
of complex many-body states under well controlled cir- 
cumstances, a situation that can be hardly found in real 
solids. 
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by the NSF grant DMR-0343790. 



FIG. 3; Many-body energy gaps at half-filling (/i = 0), as a 
function of Uff/tf, for V/tf = 3 and at a fixed value of the 
coherence length ^ = 1. Dashed line: s-wave gap; Continuous 
line: d-wave gap; Dotted-dashed line: antiferromagnetic gap. 
The inset shows a magnified plot of the d-wave regime. 



of spinless fermions. This can be obtained by suppressing 
one of the spin indices in or |(21) . In this case there is a 
major simplification in the problem since Uff is absent: 
in a spinless problem there can be only one fermion per 
site, as per Pauli's principle. Hence, in the absence of 
bosons, the spinless gas is non-interacting. The bosons, 
however, mediate the interaction between the fermions. 
Since the fermions are in different lattice sites the pair 
wavefunction has necessarily a node and hence, no s-wave 
pairing is allowed. In other words, in the spinless case 
the anti-symmetry of the wavefunction requires pairing 
in an odd angular momentum channel. In fact, we find 
that throughout the entire phase diagram the fermions 
develop p-wave pairing. At half-filling we find a similar 
behavior of CDW fluctuations on short scales, analogous 
to the flow shown in Fig. [3 One should point out that 
in real solids the conditions of " spinlessness" behavior is 
hard to achieve since it usually requires complete polar- 
ization of the electron gas, that is, magnetic energies of 
the order of the Fermi energy (a situation experimentally 
difficult to achieve in good metals). However, in cold 
atom lattices this situation can be easily accomplished 
with the correct choice of atoms. 

The many-body states discussed in this paper can be 
observed through various methods: AF order could be 
revealed in time-of-flight images and Bragg scattering 
[2^ . noise correlations 8] can be used to detect the vari- 
ous pairing phases, laser stirring experiments [3| can be 
used to detect the phase boundary between AF order and 
pairing. The short-scale CDW fluctuations should give a 
signature in a photo- association measurement. RF spec- 
troscopy 25] can be used to quantify the gaps of the 
various phases. 

In summary, we have used a functional RG approach 
to study BFM of ultra-cold atoms in a 2D optical lattice. 
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